Using Cross-validation Methods
2024-07-30
CV widely used in various fields including:
Generalizability:
How well predictive models created from a sample fit other samples from the same population.
Overfitting:
When a model fits the the underlying patterns of the training data too well.
Model fits characteristics specific to the training set:
Hyperparameters:
Are model configuration variables
Subsets the data into k approximately equally sized folds
Split The Subsets into test and training sets
Repeat k Times
Calculate the mean error
| Method | Computation | Bias | Variance |
|---|---|---|---|
| k-Fold | Lower | Intermediate | Lower |
| LOOCV | Highest | Unbiased | High |
k-fold where k = 5 or k = 10 is recommended:
By measuring the quality of fit we can select the model that Generalizes best.
\[ \text{MAE} = \frac{1}{n} \sum_{i=1}^n |y_i - \hat{f}(x_i)| \tag{1} \]
\[ \text{RMSE} = \sqrt{\frac{1}{n}\sum_{i=1}^{n}(y_i-\hat{f}(x_i))^2} \tag{2} \]
\[ \text{R}^2 = \frac{SS_{tot}-SS_{res}}{SS_{tot}} = 1 - \frac{SS_{res}}{SS_{tot}} = 1 - \frac{\sum_{i=1}^{n}(y_i - \hat{f}(x_i))^2}{\sum_{i=1}^{n}(y_i-\bar{f}(x_i))^2} \tag{3} \]
(James et al. 2013), (Hawkins, Basak, and Mills 2003), (Helsel and Hirsch 1993)
\[
CV_{(k)} = \frac{1}{k}\sum_{i=1}^{k} \text{Measuer of Errori}_i \tag{4}
\]
\[ CV_{(n)} = \frac{1}{n}\sum_{i=1}^{n} \text{Measuer of Errori}_i \tag{5} \]
Yeh modeled compression strength of high performance concrete (HPC) at various ages and made with different ratios of components.(I-C Yeh 1998)
The data is available for downloaded at UCI Machine Learning Repository.
UCI Repository HPC Data
(I-Cheng Yeh 2007).
All variables are quantitative
|
\[ \hat{Strength} = 28.258_\text{Cement + } 0.067_\text{Superplasticizer + } 0.872_\text{Age } 0.111_\text{Water} \]
| Measure of Error | Result |
|---|---|
| RMSE | 12.13 |
| MAE | 9.23 |
| R2 | 0.46 |
| Measure of Error | Result |
|---|---|
| RMSE | 12.13 |
| MAE | 9.23 |
| R2 | 0.46 |
| Measure of Error | Result |
|---|---|
| RMSE | 11.87 |
| MAE | 9.43 |
| R2 | 0.49 |
| Measure of Error | Result |
|---|---|
| RMSE | 8.73 |
| MAE | 6.82 |
| R2 | 0.73 |
| Measure of Error | Result |
|---|---|
| RMSE | 8.73 |
| MAE | 6.82 |
| R2 | 0.73 |
| Measure of Error | Result |
|---|---|
| RMSE | 5.93 |
| MAE | 4.32 |
| R2 | 0.87 |
| Measure of Error | Result |
|---|---|
| RMSE | 8.27 |
| MAE | 6.39 |
| R2 | 0.75 |
| Method | Measure of Error | Linear Regression | LightGBM |
|---|---|---|---|
| 5-Fold | RMSE | 12.13 | 8.73 |
| 5-Fold | MAE | 9.23 | 6.82 |
| 5-Fold | R2 | 0.46 | 0.73 |
| LOOCV | RMSE | 12.13 | 5.93 |
| LOOCV | MAE | 9.23 | 4.32 |
| LOOCV | R2 | 0.46 | 0.87 |
| NCV | RMSE | 11.87 | 8.27 |
| NCV | MAE | 9.43 | 6.39 |
| NCV | R2 | 0.49 | 0.75 |
All figures were created by the authors